Skip to content

Add native Bazel parity for MSTest UI packages - #12451

Open
Amaury Levé (Evangelink) wants to merge 6 commits into
dev/amauryleve/mstest-sdk-compositionfrom
dev/amauryleve/native-mstest-package-parity
Open

Amaury Levé (Evangelink) wants to merge 6 commits into
dev/amauryleve/mstest-sdk-compositionfrom
dev/amauryleve/native-mstest-package-parity

Conversation

@Evangelink

@Evangelink Amaury Levé (Evangelink) commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Pull request

Summary

Native-only final layer replacing #12435, following #7225, #12412 and #12428. Depends on SDK composition #12452, explicit packages #12450, and shared core #12446; those layers retain ownership of shipping-source, runtime, package-policy and SDK-policy changes.

Adds modern native MSTest.Uwp/MSTest.WinUI archives with canonical lib/buildTransitive paths, matching adapters/services/satellites and exact matching-core dependencies. Desktop/UWP/WinUI consumers share the ordinary native core and its core-owned, non-UI MSTest.TestFramework.Extensions forwarding facade. There is no separate facade package or UI implementation in it.

Addresses the six review threads: first-use evidence reads are guarded, modern filename joins use Path.Join, and obsolete facade-absence checks now verify forwarding and one actual core identity. The native SDK UI expectations retain the shipping SDK's packaged-app helper. The parent-relative diff is 14 files, with no top-owned src/, shipping MSBuild packaging, runtime, resource or SDK-policy changes.

The additional adapter review is fixed in 8f5e1303e3a8d906a638a1a1d0d45c8cd7d129bb: the native adapter archive supports net10/net11 only, without the accidental Windows net462 group/payload/resolver. Legacy compile labels and the independently usable ordinary core/facade remain. Native legacy adapter runtime package consumption is not supported or advertised.

The UI-isolation validator review is fixed in 1d36da4aab8f5fa4724039ef4b9684eadc71d97f: explicit conjunction parentheses keep bundled-core and cross-flavor checks independent. Six real-archive cases reproduce the old three misses and all pass with the correction; no compiled product changed.

Design and decision record

Native parity and final validation records the source states, decisions, real package/consumer checks, controlled regression experiments and limitations. The lower layers' facade, exact-identity oracle, 26/5 compatibility inventories, net462 metadata dependency, environment ownership and Markdown fixes are preserved rather than replayed as duplicate top-layer fixes.

Demonstration

This is a nonvisual archive/library contract; physical archives, loaded identities/hashes and actual runner results are more useful than a staged screenshot. The current native packages are 5.0.0-preview.bazel; each UI archive pins core to [5.0.0-preview.bazel].

Package Entries / satellites Archive SHA-256
MSTest.TestFramework including its facade 82 / 52 0FE53889979852701661F3A9E99F8C7D4A6C4A03BF53EB95B56B363F57D508AC
MSTest.Uwp 43 / 26 3918F3300F6C6A11F99D4DA5DEBE8D1EB928378EC5E78425EE8AA91A0782B2CF
MSTest.WinUI 56 / 39 5296F333ECFA30AEC3587B2F8D61CFBC8461D8B7459D6F71A6723560562BF486

Canonical UI TFMs are net10.0-windows10.0.17763 and net10.0-windows10.0.18362. Code-equivalent desktop/UWP/WinUI probes loaded the same ordinary core (D392BF09B480D19856C94B7DD47C092CA348B5B89168152D0E45FDCCE93873B1) and facade (BDF410F170B0CCA363D87A3A0BB6BB2889064D57B9C09FD861E06073F5A973B2). Real native console app-model registration/activation and both controllers also passed; this is not MSTest XAML/UI-thread application activation.

Validation and regression proof

Published head: 1d36da4aab8f5fa4724039ef4b9684eadc71d97f; SDK parent: 0afc810d4d0369a157c1461158fd6aab713825a7; agreed main snapshot 94711f3b4db336920e628039fdd54437458ae441 is an ancestor. The exact observed old-head lease was used, and both destination ref and live PR head were read back.

Current-head PR-merge/Azure CI: UNVERIFIED/BLOCKED, not all green. The successful historical native branch-head run below is separate evidence. At the post-push observation (2026-11-16 18:26 UTC), actual PR merge ref d821432b3582591a81f698d97cd346b812990f66 still bound old head bb5012718fdeb0d82bdc9eb2cd29fe106bc4e9b7, not new head 1d36da4aab8f5fa4724039ef4b9684eadc71d97f. The live PR and origin ref both match the new head, and the live SDK parent is 0afc810d4d0369a157c1461158fd6aab713825a7. The underlying stale-ref recomputation cause is not exposed. No Azure run was queued or credited from that stale ref; ending the brownout does not establish a current merge binding.

Current label workflow: FAILED/BLOCKED separately. The scheduled brownout window has elapsed. The new-head run 43928865566 / job 133515594121 started normally, then failed at 18:27 UTC with GitHub did not calculate pull request mergeability after five attempts. (exit 1). This is an actual unknown-mergeability metadata failure, not the earlier startup policy block or a product build/test failure; its guard was not relaxed. Historical 07ce label runs 43903763486 and 43904487405 remain recorded as pre-18:00 pull_request_target brownout startup failures, not new-head successes. No protection changes, approvals, comment-mutating reruns or older-head green substitution were used.

Current SDK-fixture-only parent resync

The six native commits were replayed unchanged from SDK 26aac9ff0e9b78ab2d0aebbd3eb66a30c9f6c652 onto its exact child 0afc810d4d0369a157c1461158fd6aab713825a7. All six range-diff entries are identical. The entire previous-to-current native tree delta is only test/IntegrationTests/MSTest.Acceptance.IntegrationTests/PackagedAppIntegrationTests.cs plus the existing SDK decision record; the inherited fixture patch SHA-256 is 54A5292A51756BEE5ABCF932A01C470DF18CBFA940DC85B0ECC2AEC608B1FB1C and matches the released parent byte-for-byte. Product src, native eng, utilities and unit-test trees are unchanged; the full test tree is not claimed identical.

The SDK owner records the same six selected UIDs failing at the old composition guard and passing with unchanged assertions after the real-package fixture correction, plus the full 68-case class passing with zero skips under normal parallelism (42 WinUI / 4 UWP / 22 ordinary). That is inherited SDK-owner evidence, not a native-owner rerun or a production-guard relaxation. Existing catalog tests (37), catalog matrix and both relevant Markdown records pass after replay. No native consumer/controller build, dispatch or rerun was performed merely for this unrelated integration-fixture delta.

Historical native branch-head CI: one ordinary 07ce dispatch

Run 43912280806 completed SUCCESS for exact head 07ce2a9ef15db60a385a3211ae48efa1168239cf, event workflow_dispatch, existing workflow .github/workflows/bazel.yml (406921353), branch dev/amauryleve/native-mstest-package-parity, attempt 1 / dispatch count 1. The authorized command was gh workflow run bazel.yml --repo microsoft/testfx --ref dev/amauryleve/native-mstest-package-parity; exit 0 was followed by unique actual run-ID/event/head readback, not treated alone as proof.

OS Actual job Verified result
Ubuntu 133457094786 SUCCESS: full Bazel graph/native consumers and native package/self-contained runtime gates executed successfully.
Windows 133457095024 SUCCESS: the same gates plus the expanded Windows/compatibility matrix, five CLR consumers and four additional unit suites.

Both actual checkout logs show the exact 07ce2a9ef15db60a385a3211ae48efa1168239cf source. Terminal job/step metadata binds the required validators to that run; cleanup and evidence preservation succeeded on both OSes. The Windows-only gate was skipped only on Ubuntu as intended, and main-only cache publication was skipped for this branch. Logs, step metadata and the single-dispatch audit are retained in native-07ce-branch-dispatch-proof.json and its referenced files; the run/job links above provide the hosted evidence.

Scope: historical 07ce native branch-head proof ONLY, not a new-head 1d36 run, stale PR-merge/Azure proof or overall CI green. Native/product inputs remain source-equivalent after the fixture-only resync above, but this run is not relabeled to the rewritten head. The ordinary read-only workflow did not alter protections, source, refs or workflow policy, and no retry, alternate identity/trigger, approval or comment-mutating run was used. No additional dispatch/rerun/approval is part of this evidence update.

UI-isolation proof and prior SDK-parent resync

Maintained UI-isolation record records the exact PowerShell precedence defect. The same actual validator functions run against two original UI archives and four one-DLL mutations: CoreInUwp, CoreInWinUI, UwpInWinUI, and WinUIInUwp. The old source accepts the first three invalid inputs: 6 executed, 3 passed, 3 failed, exit 1. Parenthesizing the conjunctions gives 6 executed/passed, exit 0, zero skips, with every paired input hash identical.

The validator-only patch SHA-256 is 54674D28C4E71C7CAAA87E4FEB35362104718EDA2E522CAFC0204799D025C03F; corrected validator SHA-256 is 62334CD8DFF849B02AEE69C9C1C940FB324DD25B331361ED89F206E963E68ACA. pwsh -NoLogo -NoProfile -File .\eng\bazel\validate-mstest-packages.ps1 -OutputRoot Q:\tfb\n12451-80f9f2\ui-isolation-review-4588012054\modern -IncludeWindowsPlatformPackages -VerifySatelliteSensitivity exits 0: all ten archives, fresh real net10/net11 positive/failure/empty consumers, MSBuild pass/failure and satellite removal/restoration. All ten archive hashes are identical to the adapter-boundary validation below, so existing SDK/CLR/core-only proofs remain product-equivalent, not claimed reruns.

This was tested at 08ef090a42ab79df4a47789f1bec29b54a515fb1, then replayed unchanged onto released SDK 26aac9ff0e9b78ab2d0aebbd3eb66a30c9f6c652. The ancestry replay adds only the exact thirteen lower decision-record lines; entire src, test and eng trees match the tested state, all six range-diff entries are identical, and final Markdown lint passes. No consumer/controller rebuild was run merely for that documentation resync. Paired cases/logs/hashes and replay evidence remain under Q:\tfb\n12451-80f9f2\ui-isolation-review-4588012054.

Adapter-boundary review proof

Maintained support-boundary record includes the actual consumer contract, approaches and outcomes. The adapter repair was tested as source 204d4850de60b2b2fbc28a4a1934ad73f2b2c673 plus the two-file native rules/validator patch SHA-256 86A8AA8C30892EF6C97777D863F8044C4E34F7BBBF9A9632F723C7E5B2BEAA0D; the policy is retained in rebased commit 8f5e1303e3a8d906a638a1a1d0d45c8cd7d129bb, with the later UI-only guard correction described above. Only the adapter archive changed: old 6129E376D454AC01B0EF15E1F6BCE854504AD9F558B6E8875990BA1B4991E50D, corrected 9652B932323E13E15B4C44B622A07FF642679268D5F59505824E697299A2807D. The other nine archive hashes, including core and both UI packages, are identical.

Exact command / case Result
Pinned native SDK dotnet restore LegacyConsumer.csproj --configfile NuGet.config --packages Q:\tfb\n12451-80f9f2\net462-review-4587032669\red\confirmed-packages --disable-build-servers -bl:Q:\tfb\n12451-80f9f2\net462-review-4587032669\red\restore-confirmed.binlog --nologo -v:minimal in the owned red consumer Exit 1 / NU1202, actual target net462, at native TRX abstractions supporting only net10/net11. Physical dependency nuspecs confirm the incompatible closure; four resolved native archives hash-match the producer. This reproduces the defect, not a passing legacy scenario.
pwsh -NoLogo -NoProfile -File .\eng\bazel\validate-mstest-packages.ps1 -OutputRoot Q:\tfb\n12451-80f9f2\net462-review-4587032669\modern -IncludeWindowsPlatformPackages -VerifySatelliteSensitivity Exit 0. Ten repacked archives; real net10/net11 package consumers, MSBuild parent pass/failure and satellite removal/restoration, with new cache/loaded hashes.
pwsh -NoLogo -NoProfile -File .\eng\bazel\validate-sdk-packages.ps1 -EvidenceRoot Q:\tfb\n12451-80f9f2\net462-review-4587032669\sdk -Jobs 2 Exit 0. Fourteen evaluations, five builds, actual SDK/metapackage runs and native dotnet test.
pwsh -NoLogo -NoProfile -File .\eng\bazel\windows\validate.ps1 -ArtifactsDirectory Q:\tfb\n12451-80f9f2\net462-review-4587032669\windows Exit 0. Five CLR probes, exact 26/5 inventories, 1,096 passed / 13 existing skips across four additional suites. No registration/XAML activation is claimed.
Core-only net462 consumer, exact native SDK build/runtime commands in green\core-evidence.json Build 0 / runtime 0. Actual .NETFramework,Version=v4.6.2 executable on CLR 4.0.30319.42000; both forwarded identities and core/facade producer/cache/loaded hashes match.
Corrected archive into the same actual net462 adapter restore shape, isolated green\legacy-packages Restore 0, compatible-empty. Assets contain only MSTest.TestAdapter/5.0.0-preview.bazel: { type: package }, with no dependencies, build, compile or runtime assets. The cache matches the corrected archive. No direct adapter NU1202 or restored legacy runtime support is claimed.

Focused SDK 15/15, build tools 13/13 and both ownership analysis cases pass again; 37 catalog tests, catalog matrix, Markdown lint and action-pin audit pass. Raw evidence and exact probe commands are retained locally under Q:\tfb\n12451-80f9f2\net462-review-4587032669, not represented as public artifact links. Only owned caches/outputs were used; the owned native server was shut down. No invented marker libraries, guard assets, general legacy MTP closure or lower-layer repair was added.

Earlier composed-product evidence

Earlier native package/SDK/controller validation was repeated on repaired SDK code at ad6ebd567cfe46e7f3f85de00ea3c8d760cecebb / parent a84c52f49168d02ca50b6335f7c8eb2132aa2d77. The catalog replay and evidence-only Markdown commit produced prior head ff10fec9f94ca06e4266c5178553120ae40c5db3, whose production/native code matched that tested code. The subsequent ancestry-only update to 204d4850de60b2b2fbc28a4a1934ad73f2b2c673 added only the lower OpenTelemetry E2E test's explicit processor ownership and shutdown assertion (b97623a7069f8e78794370e83aa0185293a53728, one unit-test file, +4/-3). The entire ff10-to-204 tree was verified to differ only by that exact test patch. No unrelated consumer/runtime/controller rebuild was performed merely for that unit-test change. The latest adapter rule/validator changes and freshly repeated consumers are identified separately above. Pinned Bazel 9.1.1, SDK/runtime 11.0.100-rc.3.26522.148 / 11.0.0-rc.3.26522.148, Windows x64; self-contained consumers execute actual net10/net11 runtimes.

Both native CI matrices passed on prior head ff10fec9f94ca06e4266c5178553120ae40c5db3; that and the explicit 07ce branch-head run remain historical evidence, not relabeled as the new head's CI. All checks, including automation approval/cancellation states, are assessed against actual published head 1d36da4aab8f5fa4724039ef4b9684eadc71d97f. Prior bb501/07ce automation states remain historical, not new-head successes; no comment-mutating workflows were approved or rerun.

Exact command Result
pwsh -NoLogo -NoProfile -File .\eng\bazel\validate-mstest-packages.ps1 -OutputRoot Q:\tfb\n12451-80f9f2\m-final -IncludeWindowsPlatformPackages -VerifySatelliteSensitivity Exit 0. Ten real archives; fresh net10/net11 consumers and native MSBuild invocation verify NativeAssembliesAreLoadedFromConsumer, KoreanSatellitesAreCopiedAndLoadable, PackagedTargetsProvideCoreAndParallelization; intended child exits 2/9 and MSBuild failure exit 1 are asserted, not false failures.
pwsh -NoLogo -NoProfile -File .\eng\bazel\validate-sdk-packages.ps1 -EvidenceRoot Q:\tfb\n12451-80f9f2\s-final -Jobs 2 Exit 0. Fourteen evaluations, five real consumer builds, default/minimal SDK and metapackage execution, native dotnet test, expected failure/empty exits. The four UI policy cases include the existing packaged-app helper and aligned MTP version.
pwsh -NoLogo -NoProfile -File .\eng\bazel\windows\validate-app-model.ps1 -ArtifactsDirectory Q:\tfb\n12451-80f9f2\a-final Exit 0. Actual registration/activation, suppressed launcher, net10/net11 controllers, discovery without handshake, failure/zero-result paths; 16 unique append-only provenance reports and owned registration cleanup.
.\eng\bazel.ps1 test //:sdk_package_tests //:build_tools_tests //:windows_targeting_test //:windows_asset_selection_test --jobs=2 --nocache_test_results --test_output=errors --noshow_progress --show_result=0 Exit 0. SDK 15/15 and build tools 13/13 executed/passed, zero skips; both analysis cases executed/passed.
python .github\scripts\test_mutation_testing.py; python .github\scripts\mutation_testing.py matrix --manifest eng\mutation-testing\modules.json --repository-root . Both exit 0. 37 configuration tests; 43 source projects = 32 enabled modules + 11 explicit exclusions.
markdownlint-cli2 docs\decisions\explicit-mstest-packages.md; python .github\scripts\check_action_pins.py Both exit 0. Existing Markdown rules pass; action-pin audit passes. No workflow regeneration or check suppression.

Supporting interim Windows validation also passed the full //... graph/composed-product gate and five staged CLR probes with exact 26/5 inventories, plus four additional unit suites (1,096 passed, 13 existing skips). These supporting runs are identified separately in the decision record, not mislabeled as execution of the later stack.

Raw logs/binlogs/TRX, archive/cache/loaded-bit manifests and controlled variants are retained locally under the task-owned output root; they are not public artifact URLs. The accessible command/result transcript and source/package identities are preserved here and in the maintained record. The app-model validator's existing null revision field is supplemented by a source-state record and matching captured input hashes.

Remaining scope limits: native classic UAP production, MSTest XAML/UI-thread application activation, other architectures and release-branch/backport validation are not claimed. No rollout, auto-merge or merge action is enabled. Latest-head CI is being monitored; local validation alone is not a CI-green claim.

Test-value evidence

Bound to the tested code and final code-equivalence identities above. These are controlled input/shipping-layout experiments, not pristine-parent or CI-interleaving claims.

Contract / exact case Wrong behavior or input Correct behavior
Independent UI-isolation predicates / CoreInUwp, CoreInWinUI, UwpInWinUI, WinUIInUwp and two valid controls Actual old function: 6 executed, 3 failed, exit 1, because the three reported invalid archives are accepted. Each mutation adds one real DLL without altering its original nuspec. Same six cases and archive hashes: 6 passed, exit 0, zero skips; all four invalid archives are rejected independently and both original UI archives remain valid.
Portable-only native adapter / NativeAdapterPortableLayout Same maintained validator on the original real archive: 1 executed, 1 failed, exit 1, at the exact extra-net462-group assertion. Validator SHA-256 A5D6E4C6332A26F1DDFC9FE8C9EB7384173FF73EC9067D675299A3B84DE3CE1A. Same validator on the repacked archive: 1 executed/passed, exit 0, zero skips; actual modern package/SDK/CLR consumers also pass. This is the regression proof; the incompatible restore alone is not test proof.
No leftover legacy adapter payload / NativeAdapterRejectsLegacyPayloadWithoutLegacyGroup Owned copy of the old archive with only its legacy nuspec group removed: 1 executed, 1 failed, exit 1, at the explicit legacy-payload assertion; real payload bytes retained. Fresh portable archive passes the same maintained validator. Mutation archive SHA-256 E5F9142643B952918AB0E8D56DDA2BB6EA59854D3A7F554BD34F79D51E6A3AAA; no setup/compiler failure counted.
First-use provenance / actual Run-MSBuildConsumer Same missing-index regression fails against the old unguarded read; the existing-index case remains valid. Both missing/existing-index cases pass, and the real native MSBuild pass/failure scenarios pass.
Required facade / PackagedTargetsProvideCoreAndParallelization Delete only the facade DLL in an owned copy of the freshly published final net11 consumer: 1 executed, 1 failed, exit 2, at the explicit facade-presence assertion. Compiled test DLL is unchanged. Same case 1 executed/passed, exit 0 before and after restoring the producer DLL; forwarding/core identity assertions remain intact.
Satellite flow / KoreanSatellitesAreCopiedAndLoadable Repack without the native Korean adapter satellite into a separate feed/cache: 3 executed, 1 intended failure, exit 2, at the exact missing-satellite assertion. Original and restored native consumers execute all three positive methods successfully.
SDK UI oracle / four sdk-uwp-* and sdk-winui-* evaluations Old expected set rejects the correct shipping SDK helper package. This is a test-oracle repair, not disabled production behavior. All fourteen evaluations and real composed consumers pass without suppressing helper injection.
Exact core pin, ordinary-core ownership and SDK guards Earlier output/layout mutation evidence remains recorded; no new wrong-core/guard-removal mutation is claimed for this maintenance pass. Current physical pins, exact ownership/metadata/hash checks and all 15 SDK cases pass; those unexecuted counterfactuals remain explicitly unverified.

@github-actions github-actions Bot added the state/needs-review Awaiting review from the team. label Nov 13, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧪 Test quality review — PR #12451

Compared merge base 06ee2b01d972423dd9d4f06b8ef5c8256cda9e3c with head eebd929e026ce0ada524916c6a3276a4baf4d177. Evidence applies to this head, not every intermediate commit.

Test-value evidence

Contract Test / decisive assertion Counterfactual Status
Meta-package framework set is net10.0;net11.0 Generate_MetaPackagePreservesDependencyAndAssetFlow: AreSequenceEqual on dependency groups and files targets Restoring net8.0/net9.0 in generation would fail both sequence assertions Static prediction
Meta-package rejects retired framework, UI reference, and UI condition drift Generate_RejectsContractDriftBeforeWritingOutputs rows meta-retired-framework, meta-ui-reference, meta-ui-condition (error text, outputs not written) Removing the generator guard would let generation succeed, so the expected error would not occur. The fixture strings were checked against Package.csproj content, so each replacement does apply. Static prediction
Exact-framework pin in MSTestPackageVersion.cs, UI archive layout, and Windows probes The PR's decision record cites executed evidence (Windows x64, Bazel) The record is author-reported. This workflow did not verify it. It reports no counterfactual for the flavored-core and SDK guard removals. Unverified (not executed here)
Docs, .bzl and .ps1 changes n/a Non-executable by unit tests here Not applicable

No executed paired-revision records were inspected. The static predictions above are not execution proof.

🧵 Parallel safety

STATUS: PARTIAL. The specialist subtask was not run in this pass. The changed tests use per-test temporary paths, according to the PR's own record, and I found nothing contradicting that in the diff. Parallel-safety coverage is not complete and should not be read as clean. Run /parallel-audit for a full result.

Per-test assessment

Pseudo-mutation evidence below consists of static predictions, not executed mutation results.

Result Quality Test Notes How to improve
Pass A (90–100) mod SdkPackageTests.Generate_MetaPackagePreservesDependencyAndAssetFlow Exact sequence assertions pin the new framework set and file targets. None
Pass B (80–89) mod SdkPackageTests.Generate_RejectsContractDriftBeforeWritingOutputs Three new data rows each trigger a distinct drift whose fixture replacement is present. None

No inline findings. Existing-thread deduplication was not needed because no inline comments were queued.

<sub>This advisory review is heuristic and non-blocking.

Re-run the combined review with /review-tests, or only the parallel-safety specialist with /parallel-audit.</sub>

🤖 Automated content by GitHub Copilot. Generated by the Test Quality Review on PR (on open / sync) workflow. · copilot · auto · ~23 AIC · ⌖ ~1 AIC · ⊞ 8.1K

@microsoft-github-policy-service microsoft-github-policy-service Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Nov 13, 2026
@github-actions github-actions Bot added needs/merge-conflict-resolution PR has merge conflicts that must be resolved. and removed state/needs-review Awaiting review from the team. labels Nov 13, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

🤖 Automated review by GitHub Copilot. Generated by the Expert Code Review workflow. To request a follow-up action, reply by tagging @copilot directly.

Scope: Bazel native-parity layer (eng/bazel, packaging/validation scripts, docs). No shipping src/ changes, so there is no public API, localization or IPC impact. I reviewed this statically only and executed nothing.

Finding (1 inline): an unguarded Get-Content of the provenance file in Run-MSBuildConsumer can throw if the file doesn't exist yet.

Notes, no action required:

  • windows_controller_net10 and windows_controller_net11 are used by validate-app-model.ps1 and are derived from framework.split(".")[0] in windows.bzl. I couldn't see the controller framework list in the diff, so please confirm it now includes net10/net11.
  • Test-value evidence is recorded in the decision doc. It is mutation-based, not a pristine-baseline run, and the doc says so. That seems proportionate for the build/validation-only changes.

Comment thread eng/bazel/validate-mstest-packages.ps1 Outdated
@{ Name = 'failure'; Filter = 'FullyQualifiedName~IntentionalFailure'; Exit = 1; Tests = @('IntentionalFailure'); Failed = 1; Message = 'BAZEL_INTENTIONAL_FAILURE' }
)) {
$name = "msbuild-$($scenario.Name)"
$previousEvidence = @(Get-Content -LiteralPath $env:TESTFX_BAZEL_CONSUMER_EVIDENCE).Count

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Get-Content -LiteralPath throws under $ErrorActionPreference = 'Stop' when the evidence file doesn't exist yet. Run-Consumer guards the same read with Test-Path, but this one doesn't. The file is only created by the first successful pass scenario, so this would fail on the first iteration unless an earlier consumer already wrote to the same TESTFX_BAZEL_CONSUMER_EVIDENCE path. I couldn't confirm from the diff that the MSBuild consumer shares that path, so please check. Use the same Test-Path ... ? count : 0 guard to make it independent of ordering.

@Evangelink Amaury Levé (Evangelink) Nov 16, 2026 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Guarded the first evidence-index read with Test-Path in 7e998e058cb8c89eefb099b97723089ff2bd2448. The actual Run-MSBuildConsumer regression now handles both missing and existing indexes, and the real native MSBuild pass/failure run passes.

Auto-replied by the GitHub Copilot app

@microsoft-github-policy-service microsoft-github-policy-service Bot added the state/needs-review Awaiting review from the team. label Nov 13, 2026
@github-actions github-actions Bot added needs/merge-conflict-resolution PR has merge conflicts that must be resolved. state/needs-review Awaiting review from the team. and removed needs/merge-conflict-resolution PR has merge conflicts that must be resolved. state/needs-review Awaiting review from the team. labels Nov 13, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧪 Test quality review — PR #12451

Compared merge base d8a16d7eaaa55a5ce070cfa29033cec32fcdfc3e with head 78ddb111a01b180f6ab61a0296d4d7ff0ea2688a. Evidence applies to this head, not every intermediate commit.

Test-value evidence

Contract Test / decisive assertion Counterfactual Status
Meta-package limited to net10.0/net11.0 Generate_MetaPackagePreservesDependencyAndAssetFlow sequence assertions on groups and files Generating net8/net9 groups would fail the sequence assertions Static prediction
Meta-package rejects retired-framework, UI reference, and flipped UWP condition Generate_RejectsContractDriftBeforeWritingOutputs rows meta-retired-framework, meta-ui-reference, meta-ui-condition Removing the projectDependencies path/condition check in SdkPackageTool.cs lets them pass. Each fixture mutation matches a real ProjectReference line, so the replacements apply. Static prediction
--exact-framework UI pin in MSTestPackageVersion.cs, Bazel rules, validators, native probes No changed unit test; the PR's decision document reports executed Bazel/Windows evidence n/a Unverified; the author's evidence is not accessible and matches no exact revision. Not exercised by the changed unit tests.

No executed paired red/green records are accessible. Predictions are not counted as mutation scores.

🧵 Parallel safety

Parallel-safety analysis was not completed as a separate specialist run in this review (STATUS: PARTIAL). The only changed test class creates per-test temp paths via PathFor and Write, and this is not a clean result. No parallelization configuration changed.

Per-test assessment

Pseudo-mutation evidence below consists of static predictions, not executed mutation results.

Result Quality Test Notes How to improve
Pass B (80–89) mod SdkPackageTests.Generate_MetaPackagePreservesDependencyAndAssetFlow Exact framework and file sequences protect the narrowed contract. None
Pass B (80–89) mod SdkPackageTests.Generate_RejectsContractDriftBeforeWritingOutputs Three new data rows each trigger the unsupported-reference error. None

No inline findings. Deduplication was not an issue because no inline candidates were prepared.

<sub>This advisory review is heuristic and non-blocking.

Re-run the combined review with /review-tests, or only the parallel-safety specialist with /parallel-audit.</sub>

🤖 Automated content by GitHub Copilot. Generated by the Test Quality Review on PR (on open / sync) workflow. · copilot · auto · ~30 AIC · ⌖ ~1 AIC · ⊞ 8.1K

@microsoft-github-policy-service microsoft-github-policy-service Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Nov 13, 2026
@github-actions github-actions Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Nov 13, 2026
Comment thread eng/bazel/windows/AppModelHost.cs Fixed
Comment thread eng/bazel/windows/AppModelHost.cs Fixed
Comment thread eng/bazel/windows/LibraryProbe.cs Fixed
@microsoft-github-policy-service microsoft-github-policy-service Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Nov 13, 2026
@github-actions github-actions Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Nov 13, 2026
@microsoft-github-policy-service microsoft-github-policy-service Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Nov 13, 2026
@github-actions github-actions Bot added state/approved Proposal approved; ready for implementation. and removed state/needs-review Awaiting review from the team. labels Nov 13, 2026
@github-actions github-actions Bot added the state/approved Proposal approved; ready for implementation. label Nov 16, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧪 Test quality review — PR #12451

Compared merge base b3e00771805ce47afb4404a68edd4184782099be with head 204d4850de60b2b2fbc28a4a1934ad73f2b2c673. Evidence applies to this head, not every intermediate commit.

Test-value evidence

Contract Test / decisive assertion Counterfactual Status
SDK meta-package generator rejects retired framework, UI project references and flipped UI condition Generate_RejectsContractDriftBeforeWritingOutputs rows meta-retired-framework, meta-ui-reference, meta-ui-condition: expected error text and no outputs written Accepting those references or conditions in SdkPackageTool.cs (the Unsupported meta-package project reference or condition check) would fail these rows Static prediction; no executed records for these exact revisions accessible (the PR's decision doc reports an author-run 15-pass TRX, which is not paired execution proof)
Bazel/PowerShell validators, .bzl rules, native probes and docs Existing Windows-only validators and probes Not executable here Unverified / not assessed (Windows native, not run by this read-only workflow)

🧵 Parallel safety

STATUS: not separately delegated. The changed tests only add data rows to an existing method; no parallelization configuration changed. Per-test paths are already used via PathFor and the PR doc states method-level scheduling with per-test temp paths. Coverage is limited to this static check, not a full specialist audit.

Per-test assessment

Pseudo-mutation evidence below consists of static predictions, not executed mutation results.

Result Quality Test Notes How to improve
Pass A (90–100) mod SdkPackageTests.Generate_RejectsContractDriftBeforeWritingOutputs Three new rows assert exact rejection message and no outputs written. None

Pseudo-mutation: removing the project-reference or condition validation → Likely killed (inferred) by the new rows. Note: all three rows share one expected message, so they do not distinguish which check rejected them.

<sub>This advisory review is heuristic and non-blocking.

Re-run the combined review with /review-tests, or only the parallel-safety specialist with /parallel-audit.</sub>

🤖 Automated content by GitHub Copilot. Generated by the Test Quality Review on PR (on open / sync) workflow. · copilot · auto · ~29 AIC · ⌖ ~1 AIC · ⊞ 8.1K

@microsoft-github-policy-service microsoft-github-policy-service Bot added the state/needs-review Awaiting review from the team. label Nov 16, 2026
@github-actions github-actions Bot added state/approved Proposal approved; ready for implementation. and removed state/needs-review Awaiting review from the team. state/approved Proposal approved; ready for implementation. labels Nov 16, 2026
Copilot AI balanced review requested due to automatic review settings November 16, 2026 15:39
@github-actions github-actions Bot added state/approved Proposal approved; ready for implementation. and removed state/approved Proposal approved; ready for implementation. labels Nov 16, 2026
@microsoft-github-policy-service microsoft-github-policy-service Bot added the state/needs-review Awaiting review from the team. label Nov 16, 2026
@github-actions github-actions Bot added state/approved Proposal approved; ready for implementation. and removed state/needs-review Awaiting review from the team. state/approved Proposal approved; ready for implementation. labels Nov 16, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧪 Test quality review — PR #12451

Compared merge base b3e00771805ce47afb4404a68edd4184782099be with head bb5012718fdeb0d82bdc9eb2cd29fe106bc4e9b7. Evidence applies to this head, not every intermediate commit. Static analysis only; no code was executed.

Test-value evidence

Contract Test / decisive assertion Counterfactual Evidence
Meta-package rejects the retired TestFramework.Extensions project reference Generate_RejectsContractDriftBeforeWritingOutputs("meta-retired-framework"): non-zero exit, Unsupported meta-package project reference, no outputs written Allow-list accepts the retired project Static prediction: likely killed. Exact production allow-list (SdkPackageTool.cs:208) was not re-traced for each path.
Meta-package rejects the WinUI reference in place of Uwp ...("meta-ui-reference"), same assertions Allow-list accepts the WinUI project Static prediction: likely killed
Meta-package rejects an inverted Uwp condition ...("meta-ui-condition"), same assertions Condition check is dropped Static prediction: likely killed
eng/bazel/*.bzl, .ps1, native probes, and the decision doc Not unit-test contracts n/a Not applicable to this review. The decision doc reports executed Windows/Bazel evidence, but no records were accessible for the exact head. Treat that as unverified.

The new --minimum-expected-tests 15 guard matches the 12 existing + 3 new rows. No paired red/green execution records were available, so there is no executed proof.

🧵 Parallel safety

STATUS: CLEAN. Assembly Bazel.SdkPackages.UnitTests uses default scheduling. The new rows only touch per-test TestContext.TestTempDirectory paths, with no static or shared state. Counts: Critical=0, High=0, Warning=0, Info=0.

Per-test assessment

Pseudo-mutation evidence below consists of static predictions, not executed mutation results.

Result Quality Test Notes How to improve
Pass A (90–100) mod SdkPackageTests.Generate_RejectsContractDriftBeforeWritingOutputs New data rows assert non-zero exit, the specific error text and that no outputs are written. None

No inline comments: no verified, actionable findings.

<sub>This advisory review is heuristic and non-blocking.

Re-run the combined review with /review-tests, or only the parallel-safety specialist with /parallel-audit.</sub>

🤖 Automated content by GitHub Copilot. Generated by the Test Quality Review on PR (on open / sync) workflow. · copilot · auto · ~50 AIC · ⌖ ~1 AIC · ⊞ 8.1K

@microsoft-github-policy-service microsoft-github-policy-service Bot added the state/needs-review Awaiting review from the team. label Nov 16, 2026
@github-actions github-actions Bot added state/approved Proposal approved; ready for implementation. and removed state/needs-review Awaiting review from the team. state/approved Proposal approved; ready for implementation. labels Nov 16, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Incorrect Boolean grouping allows the package-isolation validator to accept bundled core DLLs in either UI package and accidental UWP DLLs in MSTest.WinUI.

1 open issue
1 resolved since last review

Review effort: Balanced

Comment thread eng/bazel/validate-mstest-packages.ps1 Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Maintainer confirmation is needed because the Windows native UI archives cross NuGet, Bazel, and CLR boundaries without inspected package-consumption validation or accessible matching-revision runtime evidence.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Consume the core-owned non-UI compatibility facade, retain ordinary shared-core identities, and make provenance indexing safe on first use.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Retain the shipping SDK helper package in the four packaged UWP/WinUI evaluation cases and record the actual native validation results.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Bind real package, SDK and controller checks to the repaired source tree and record the metadata-only final parent equivalence.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Remove the accidental net462 archive group and resolver payload without changing legacy compile targets or the ordinary core/facade. Enforce the package boundary and record real restore, package, SDK and CLR evidence.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Parenthesize the flavor-specific conjunctions so PowerShell does not suppress bundled-core and cross-flavor archive checks. Record same-input real-archive red/green proof and the passing maintained package gate.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Windows package/runtime validation for head 1d36da4 remains unverified because hosted CI covers only historical 07ce2a9 and local regression/provenance artifacts are unavailable for inspection.

No open issues 🎉

Review effort: Balanced

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

state/approved Proposal approved; ready for implementation. state/needs-review Awaiting review from the team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants